{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Working With Multiple Layers\n",
    "\n",
    "First we start with loading the necessary libraries and resetting the computational graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import os\n",
    "from tensorflow.python.framework import ops\n",
    "ops.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Create a Graph Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Create Tensors\n",
    "\n",
    "Here we will create a small image of size 4x4 pixels and propagate it through multiple layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Create a small random 'image' of size 4x4\n",
    "x_shape = [1, 4, 4, 1]\n",
    "x_val = np.random.uniform(size=x_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Create the Data Placeholder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x_data = tf.placeholder(tf.float32, shape=x_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### First Layer: Moving Window (Convolution)\n",
    "\n",
    "Our first layer will be a spatial moving window of size [2x2] with stride 2 (in both height and width directions)\n",
    "\n",
    "To make this a moving window average, the value of the filter will be all 0.25."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Create a layer that takes a spatial moving window average\n",
    "# Our window will be 2x2 with a stride of 2 for height and width\n",
    "# The filter value will be 0.25 because we want the average of the 2x2 window\n",
    "my_filter = tf.constant(0.25, shape=[2, 2, 1, 1])\n",
    "my_strides = [1, 2, 2, 1]\n",
    "mov_avg_layer= tf.nn.conv2d(x_data, my_filter, my_strides,\n",
    "                            padding='SAME', name='Moving_Avg_Window')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Second Layer: Custom\n",
    "\n",
    "Our second layer will be a custom layer.  Given an input, x, this layer flattens out x and computes sigmoid(Ax+b).  Here, A and b will be predetermined constants.\n",
    "\n",
    "We then add the custom layer to the graph under the name 'Custom_Layer'.  This is for visualizing the graph in Tensorboard later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Define a custom layer which will be sigmoid(Ax+b) where\n",
    "# x is a 2x2 matrix and A and b are 2x2 matrices\n",
    "def custom_layer(input_matrix):\n",
    "    input_matrix_sqeezed = tf.squeeze(input_matrix)\n",
    "    A = tf.constant([[1., 2.], [-1., 3.]])\n",
    "    b = tf.constant(1., shape=[2, 2])\n",
    "    temp1 = tf.matmul(A, input_matrix_sqeezed)\n",
    "    temp = tf.add(temp1, b) # Ax + b\n",
    "    return(tf.sigmoid(temp))\n",
    "\n",
    "# Add custom layer to graph\n",
    "with tf.name_scope('Custom_Layer') as scope:\n",
    "    custom_layer1 = custom_layer(mov_avg_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Run Output\n",
    "\n",
    "The output should be an array that is 2x2, but size (1,2,2,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[[ 0.39856768]\n",
      "   [ 0.85437918]]\n",
      "\n",
      "  [[ 0.2236276 ]\n",
      "   [ 0.35883212]]]]\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(mov_avg_layer, feed_dict={x_data: x_val}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "After custom operation, size is now 2x2 (squeezed out size 1 dims), see below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.8636359   0.92904055]\n",
      " [ 0.78113878  0.77243596]]\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(custom_layer1, feed_dict={x_data: x_val}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Save summaries for viewing in Tensorboard:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "merged = tf.summary.merge_all(key='summaries')\n",
    "\n",
    "if not os.path.exists('tensorboard_logs/'):\n",
    "    os.makedirs('tensorboard_logs/')\n",
    "\n",
    "my_writer = tf.summary.FileWriter('tensorboard_logs/', sess.graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "![multiple_layers_tensorboard](https://github.com/nfmcclure/tensorflow_cookbook/raw/master/02_TensorFlow_Way/images/03_Multiple_Layers.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
